Parallel processing is the ability to partition a single large computer program into smaller pieces that can be executed simultaneously in different processors to achieve higher throughput. There are two general classes of parallel-processing computers: (1) the Single-Instruction/Multi-Data Stream (SIMD) computer which can apply a single instruction to multiple data elements simultaneously, such as found in array processors, vector processors and systolic computers, and (2) Multi-Instruction/Multi-Data Stream (MIMD) computers, which can apply multiple instructions to multiple data elements simultaneously.
True MIMD parallel-processing computers are rare. Many MIMD computers that are said to implement parallel processing are in fact distributed-processing computers that perform many small and near-independent processes. Other MIMD computers are redundant-processing units that are designed for fault tolerance rather than speed.
Distributed-processing computers claiming to be parallel-processing computers face problems arising from their use of a von Neumann architecture at the system level. The distributed approach is effective only as long as a computer's workload be split into different and independent processes, such as ordinary time-sharing or transaction processing.
If many interprocess communications are needed, the common global bus route to shared global memory becomes a significant constraint. With reference to FIG. 1, the global von Neumann bottleneck of distributed-processing computers is demonstrated. A global bus 10 is used to link several different processors 12 (two shown) and a global memory 14. Although each processor is accessing a shared global memory, the global memory can serve only one processor at a time. A hardware or software arbitrator must determine the priorities of the pending access requests, queue them, and serve them sequentially one request at a time. Global bus architecture causes significant delays when handling job streams with interspersed large processes that cannot be broken up. In a case such as this, the multiple processors become functionally equivalent to a single processor.
To avoid some of these problems, an MIMD computer may have multiple local processors with local memories coupled together by a interconnection network through which inputs and outputs to the processors and all the inter-processor messages are routed. Some MIMD computers may have their processors accessing both local and shared global memories, while others incorporate expensive multi-port memories to alleviate the problem of access to the memory. With whatever system is used, however, the program execution time in resolving global memory conflicts is unpredictable and may be inefficient.